Skip to content

Fix import-data and export-data correctness and error reporting - #18828

Open
Caideyipi wants to merge 3 commits into
apache:masterfrom
Caideyipi:fix/data-import-export-correctness
Open

Caideyipi wants to merge 3 commits into
apache:masterfrom
Caideyipi:fix/data-import-export-correctness

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

import-data and export-data can silently produce incorrect output or report success after an import fails. For example, a tree query containing interleaved columns from two devices currently exports only the first device and assigns it a value from the other device. Multiple custom table queries also overwrite the same output file and can use the wrong destination table.

This change fixes the data tools by:

  • Preserving the original column indices for every tree device, omitting absent values, and splitting SQL files by the number of generated INSERT statements.
  • Giving each custom table query its own output files and deriving the destination table with the existing relational SQL parser. SQL/TsFile queries with multiple source relations or a different database fail before writing. Table and column identifiers are quoted.
  • Escaping SQL text literals and emitting SQL NULL for missing values.
  • Honoring typed CSV export headers and retaining the original CSV header spelling when importing table data with case-insensitive names and types.
  • Clearing CSV batches only after successful insertion and propagating exhausted retries, worker errors, and archive errors to a nonzero process exit code.
  • Archiving each imported TsFile under its relative filename with its .resource and .mods sidecars, and reporting file-operation failures accurately.
  • Preserving the Java process exit code in the Windows import/export launchers.

Validation

  • mvn -o test -pl iotdb-client/cli -Dtest=DataToolRegressionTest,ImportTsFileOperationTest,WriteDataFileTest -Ddevelocity.off=true: 21 tests passed, including 18 new regression tests.
  • mvn -o test-compile -DskipTests -Ddevelocity.off=true: full 53-module reactor passed.
  • mvn -o test-compile -P with-zh-locale -DskipTests -Ddevelocity.off=true: full 53-module Chinese-locale reactor passed.
  • Windows launcher probes: both scripts returned Java exit code 17 with inherited ret_code unset, 0, and 23.
  • git diff --check passed.

No live-cluster integration tests were run.

This PR has:

  • been self-reviewed.
  • added regression tests for the corrected behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant